Fix rewind deadlock when Task.all had multiple failed siblings (#299) by YunchuWang · Pull Request #300 · microsoft/durabletask-js

YunchuWang · 2026-07-14T01:45:32Z

Summary

Fixes a rewind deadlock where a nested orchestration whose Task.all fan-out contained more than one failed activity never completes after a server-side (Azure Storage / DurableTask.Core) rewind. The worker re-dispatched only one previously-failed sibling and left the other as a permanently-unresolved pending task, so the child's Task.all never resolved, no SubOrchestrationInstanceCompleted was emitted, and the parent/root hung until the 30s timeout.

Closes #299

Root cause (verified)

On the Azure Storage backend, rewind is computed server-side by DurableTask.Core — this is a plain replay of a backend-rewritten history, not the EXECUTIONREWOUND / buildRewindResult path from #296 (0 ExecutionRewound events observed).

The rewind (DurableTask.AzureStorage RewindHistoryAsync / DurableTask.Core ProcessRewindOrchestrationDecision) removes/morphs a failed activity's TaskScheduled only when a TaskFailed event exists for it. Because Task.all fails fast, the child completes-failed after the first sibling failure, so later siblings' TaskFailed events are never committed (late completion for a terminal instance is dropped). After rewind:

Sibling A (TaskFailed present): its TaskScheduled is morphed away → its pending action survives replay → re-dispatched cleanly. ✅
Sibling B (TaskFailed missing): its TaskScheduled is left bare (no terminal) → handleTaskScheduled deletes its pending action assuming a completion will arrive, but after a rewind none ever does → orphaned pending task → Task.all deadlocks. ❌

Observed worker trace (single child instance): Waiting for 2 task(s) … Returning 1 action(s) (one sibling re-dispatched, runs, succeeds) then Waiting for 1 task(s) … Returning 0 action(s) = deadlock.

Fix (worker executor)

Detect a rewind revival from the NEW events. Azure Storage wakes a rewound instance with a GenericEvent (RewindTaskOrchestrationAsync sends new GenericEvent(-1, reason)); the DTS jump-start path uses ExecutionRewound. Gating on new events scopes reconciliation to the revival work item, so ordinary post-rewind replays (whose new events are normal completions) never re-dispatch genuinely in-flight tasks.
Re-dispatch orphaned activities. During replay, archive each consumed scheduleTask action. At the end of a revival replay, for every still-pending task whose scheduleTask action was consumed by a bare TaskScheduled but has no live pending action, re-add the archived action. General for N failed siblings. Sub-orchestrations are intentionally excluded (their retained SubOrchestrationInstanceCreated is revived separately by the backend).
Make duplicate TaskScheduled idempotent. DurableTask.Core ProcessScheduleTaskDecision always appends a fresh TaskScheduled per action (no dedup), so re-dispatching the bare orphan produces a second TaskScheduled with the same id. A repeat for an already-handled id is now treated as idempotent instead of raising NonDeterminismError.

57 lines of source across 2 files (orchestration-executor.ts, runtime-orchestration-context.ts).

Test evidence

New deterministic executor-level regression test test/rewind-multi-sibling-deadlock.spec.ts drives the executor through episodes modelling the classic Azure Storage backend (append-only TaskScheduled, revival GenericEvent trigger):

re-dispatches ALL previously-failed Task.all siblings after a rewind and completes — red without the fix (deadlock), green with it.
re-dispatches N (=3) failed siblings — generality check.
does NOT re-dispatch an in-flight bare TaskScheduled during ordinary (non-rewind) replay — no-regression guard (passes on main and with the fix).

Red/green proof (identical test code):

# source reverted (main behavior):
× re-dispatches ALL previously-failed Task.all siblings ...   (deadlocked: expected false, got true)
× re-dispatches N (=3) failed siblings ...
√ does NOT re-dispatch an in-flight bare TaskScheduled ...
Tests: 2 failed, 1 passed

# with fix:
√ re-dispatches ALL previously-failed Task.all siblings ...
√ re-dispatches N (=3) failed siblings ...
√ does NOT re-dispatch an in-flight bare TaskScheduled ...
Tests: 3 passed

Sister-SDK parity (corroboration)

The same defect exists in durabletask-python (independently verified against microsoft/durabletask-python@e7b5f15), which confirms this is a core, cross-SDK replay-contract issue rather than a JS-specific quirk:

Its taskScheduled handler (worker.py ~L2196-2205) pops the pending action on a bare TaskScheduled and raises a non-determinism error if the action is already gone — action = ctx._pending_actions.pop(task_id, None); if not action: raise _get_non_determinism_error(...) — while only reading (.get) the pending task. So a rewound-away sibling leaves an orphaned pending task, exactly as in JS.
get_actions() (worker.py ~L1549) returns list(self._pending_actions.values()), so a popped orphan is never re-dispatched → identical Task.all deadlock after a multi-failed-sibling rewind.
Python's rewind support (_build_rewind_result) only short-circuits the DTS jump-start path (an executionRewound event in the new events). A classic Azure Storage server-side rewind (no executionRewound) falls through to normal replay — the same path that deadlocks here — and there is no _consumed / rewind-revival re-dispatch and no idempotent duplicate-TaskScheduled guard. The durabletask-python functions-support PR Andystaples/add functions support durabletask-python#155 only added a genericEvent ignore, not re-dispatch, so the deadlock remains unfixed there.

This independently corroborates two things about this PR:

The root cause is at the DurableTask replay-contract level, not a JS bug.
The idempotent handled-id guard is load-bearing. Python's if not action: raise NonDeterminism on a repeat TaskScheduled is exactly what a naive re-dispatch would trip — because re-dispatching the orphan makes the backend append a second TaskScheduled for the same id. Re-dispatch without the idempotency guard would convert the deadlock into a NonDeterminismError, which is why this PR pairs re-dispatch with idempotent duplicate-TaskScheduled handling.

A durabletask-python parity fix is therefore warranted and should be tracked separately.

E2E status on real Azure Storage (verified)

Run on a live Azure Storage backend (fix logic ported onto the #282 gRPC-worker core; fix anchors byte-identical; callEntities=false, single target framework to force one deterministic instance):

The NonDeterminism / duplicate-TaskScheduled assumption is CLOSED — zero NonDeterminismError across numFailures 1 and 2.
On the path DurableTask.Core actually rewinds, the signaled child logs Waiting for 2 → Returning 2 (both failed activity siblings re-dispatched) and completes, whereas the un-fixed baseline logs Returning 1 and deadlocks — the decisive proof this fix is real and necessary.

This fix is necessary but not sufficient to turn RewindOrchestratorTests.RewindFailedOrchestration_ShouldSucceed(1|2) green, because that E2E also exercises a distinct, deeper gap: the root Task.all fails fast on the first sub-orchestration failure, so the second failing sub-orch's failure is dropped (late completion for a now-terminal parent). Server-side rewind therefore signals only the committed-failed subtree; the other failing subtree is re-driven with no rewind signal, so its grandchild hits the identical #299 orphan that the (correctly gate-guarded) reconcile cannot repair without a signal. Which subtree is dropped is a race. That unsignaled-sibling gap is tracked separately in #301.

Gates

npm run build — clean (all workspaces)
npm run lint — clean
npm run test:unit — 1283 passed, 0 failed (1122 core incl. the new spec and the Add rewind support #296 rewind-e2e suite, 106 azuremanaged, 55 export-history)

Notes / honesty

Root cause and fix were verified against the real DurableTask.Core / DurableTask.AzureStorage rewind source, the failing-run worker log, and the issue Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 A/B evidence.
The shipped correctness proof is the executor-level regression test, which faithfully models the append-only backend and the duplicate-TaskScheduled behavior the idempotent guard handles. The full extension E2E was additionally run live on Azure Storage (see E2E status above): on the path DTFx actually rewinds it confirms the Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 fix — the signaled grandchild logs Waiting for 2 → Returning 2 (both failed siblings re-dispatched) and completes, versus the NOFIX baseline's Returning 1 deadlock — with zero NonDeterminismError across every run, so the idempotent handled-id guard holds on the real backend. However, clean single-target-framework runs of RewindFailedOrchestration_ShouldSucceed(1|2) still fail at RewindOrchestratorTests.cs:50 for a distinct, deeper reason — an unsignaled re-driven sibling subtree, tracked in Rewind after a multi-failed-sibling WhenAll deadlocks the UNSIGNALED sibling subtree (emit-once replay can't reconcile a re-driven orphan with no rewind signal) #301 — and an earlier apparent pass was a multi-TFM (net8.0;net10.0) concurrency artifact, not a reliable green. Fix rewind deadlock when Task.all had multiple failed siblings (#299) #300 stays correctly scoped to the Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 signaled-path fix.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

After a server-side (Azure Storage / DurableTask.Core) rewind, a nested orchestration whose Task.all fan-out contained more than one failed activity would deadlock: only one previously-failed sibling was re-dispatched and the other became a permanently-unresolved pending task, so the child's Task.all never resolved and the parent/root never completed (30s timeout). Root cause: the rewind removes a failed activity's TaskScheduled only when a TaskFailed event exists for it. Task.all fails fast, so after the first sibling failure the child completes-failed and later siblings' TaskFailed events are never committed. Their TaskScheduled is therefore left BARE (no terminal). On replay handleTaskScheduled deletes the pending action assuming a completion will arrive, but after a rewind none ever does -> the task is orphaned. Fix (worker executor): - Detect a rewind revival from the NEW events (Azure Storage wakes the instance with a GenericEvent; DTS uses ExecutionRewound). Gating on new events keeps reconciliation scoped to the revival work item so ordinary post-rewind replays never re-dispatch genuinely in-flight tasks. - Archive each consumed scheduleTask action during replay; at end of a revival replay, re-dispatch every orphaned activity (pending task whose scheduleTask action was consumed by a bare TaskScheduled but has no live pending action). General for N failed siblings. - Make a duplicate TaskScheduled for an already-handled id idempotent, since the re-dispatch causes DurableTask.Core (no dedup) to append a second TaskScheduled with the same id. Adds a deterministic executor-level regression test that models the classic Azure Storage backend (append-only TaskScheduled) and is red without the fix / green with it, plus a no-regression test asserting an in-flight bare TaskScheduled during ordinary replay is not re-dispatched. Closes #299 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Fixes a worker-side replay deadlock after server-side rewinds (Azure Storage / DurableTask.Core) when a nested orchestration’s Task.all fan-out had multiple failed activity siblings, leaving one or more siblings as bare TaskScheduled events with no terminal completion and causing the child (and therefore parent) orchestration to hang.

Changes:

Adds rewind-revival reconciliation that re-dispatches “orphaned” activity schedule actions whose TaskScheduled was consumed during replay but never receives a terminal event post-rewind.
Makes duplicate TaskScheduled events for the same task ID idempotent during replay to tolerate Azure Storage’s append-only scheduling behavior.
Adds deterministic executor-level regression tests modeling the Azure Storage backend rewind shape and duplicate TaskScheduled behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
packages/durabletask-js/src/worker/orchestration-executor.ts	Rewind-revival detection + orphaned activity re-dispatch reconciliation; idempotent handling for duplicate `TaskScheduled`.
packages/durabletask-js/src/worker/runtime-orchestration-context.ts	Adds executor bookkeeping to remember consumed activity actions and handled taskScheduled IDs.
packages/durabletask-js/test/rewind-multi-sibling-deadlock.spec.ts	New executor-level regression suite reproducing and guarding the multi-failed-sibling rewind deadlock scenario.

+  private isRewindRevival(newEvents: pb.HistoryEvent[]): boolean {
+    return newEvents.some((e) => e.hasExecutionrewound() || e.hasGenericevent());
+  }


+      const impl = activityImpls[name];
+      try {
+        const output = impl ? impl(input) : undefined;
+        committed.push(newTaskCompletedEvent(id, output !== undefined ? JSON.stringify(output) : undefined));
+      } catch (err) {
+        committed.push(newTaskFailedEvent(id, err as Error));
+      }


YunchuWang · 2026-07-14T16:46:22Z

Closing in favor of a root-cause fix (tracked in #301).

Why: Analysis established that this PR's worker-side archive + revive is a band-aid that (1) only repairs the orphan on the rewind-signaled path and provably cannot fix the unsignaled re-driven sub-orchestration case (#301), and (2) becomes dead code once the real cause is removed.

The real cause is WhenAllTask fail-fast: onChildCompleted completes the whenAll on the first child failure, so a multi-failed-sibling whenAll lets the orchestration go terminal before the later siblings' TaskFailed events are committed. Those siblings keep a bare TaskScheduled with no completion, and on rewind the worker pops the pending action but no completion ever arrives -> the whenAll waits forever -> deadlock.

Cross-SDK evidence: .NET (System.Threading.Tasks.Task.WhenAll) and Java (CompletableFuture.allOf) both wait for all children and aggregate exceptions — they never drop a sibling failure and are structurally immune to this bug. durabletask-js and durabletask-python diverged by adding fail-fast.

The fix is to align durabletask-js WhenAll to wait-all semantics (complete only after every child reaches a terminal state, even on failure). That removes the orphan at the source — at every nesting level, signaled or not — and makes this revive machinery unnecessary. A replacement PR implementing the wait-all fix (JS-only) is being opened and will reference #301. durabletask-python shares the same fail-fast root cause and will be addressed separately.

YunchuWang · 2026-07-14T17:10:56Z

Replacement PR is now open: #302 (Fixes #301) — the wait-all root-cause fix described above. It changes one source file (when-all-task.ts) to complete WhenAll only after all children are terminal, surfacing the first failure without aggregation, and updates the existing fail-fast tests plus adds a two-batch multi-failed-sibling regression test for #301. No archive/revive machinery.

Copilot AI review requested due to automatic review settings July 14, 2026 01:45

Copilot started reviewing on behalf of YunchuWang July 14, 2026 01:46 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

YunchuWang mentioned this pull request Jul 14, 2026

Rewind after a multi-failed-sibling WhenAll deadlocks the UNSIGNALED sibling subtree (emit-once replay can't reconcile a re-driven orphan with no rewind signal) #301

Closed

YunchuWang closed this Jul 14, 2026

YunchuWang mentioned this pull request Jul 14, 2026

Fix #301: WhenAll wait-all + aggregate-exception semantics (root cause of rewind deadlock) #302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix rewind deadlock when Task.all had multiple failed siblings (#299)#300

Fix rewind deadlock when Task.all had multiple failed siblings (#299)#300
YunchuWang wants to merge 1 commit into
mainfrom
yunchuwang-fix-rewind-nested-taskall-multi-failed-s

YunchuWang commented Jul 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

YunchuWang commented Jul 14, 2026

Uh oh!

YunchuWang commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

YunchuWang commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause (verified)

Fix (worker executor)

Test evidence

Sister-SDK parity (corroboration)

E2E status on real Azure Storage (verified)

Gates

Notes / honesty

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

YunchuWang commented Jul 14, 2026

Uh oh!

YunchuWang commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

YunchuWang commented Jul 14, 2026 •

edited

Loading